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(54) Apparatus and method for automatic color correction 



(57) In a color correction apparatus 1 , a representa- 
tive color extracting unit 13 reads from an object color 
information memory 12 a hue distributable region of an 
object selected by an object selecting unit 1 1 and the 
distribution frequency in each split hue region to extract 
a representative color. A color correction parameter 
determining unit 15 determines from the memory con- 
tent of a color correction parameter memory 14 opti- 
mum color correction parameters adapted to the 
representative color extracted by the representative 
color extracting unit 13. A color correction processing 
unit 16 carries out, upon an input image, color correc- 
tion acting only upon the representative color of the 
object and neighborhood colors therearound. 
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D script! on 

Packground of the Invention; 

[0001] This invention relates to an apparatus and a 
method for automatic color correction and a recording 
medium storing a control program therefor and, in par- 
ticular, to a color correction technique for a color image 
in a color image processing apparatus, such as a digital 
still camera, a scanner, a display, and a printer, and in 
color image processing software. 
[0002] In color reproduction of a color image, it is 
important to realize not only exact color reproduction 
among color image processing apparatuses but also 
favorable color reproduction convenient for human 
beings. Natural objects such as flesh or skin color, blue 
sky, and green vegetation tend to attract attention of 
human beings and therefore require favorable color 
reproduction. 

[0003] Particularly for the skin color, proposal has 
been made of various skin color correction techniques. 
For example, Japanese Unexamined Patent Publication 
(JP-A) No. H08-79549 discloses the technique of 
improving the reproducibility of the skin color. In the dis- 
closed technique, a high frequency component of a spa- 
tial frequency is eliminated for a skin color region to 
lessen the granular texture and to prevent gradation or 
color missing at a highlighted portion. 
[0004] Japanese Unexamined Patent Publication 
(JP-A) No. H1 1 -1 7969 discloses a technique of correct- 
ing an unfavorable skin color deviated from a favorable 
skin-color range into a favorable skin color. In this tech- 
nique, an input image is displayed on a color CRT 
(Cathode Ray Tube). As an object color to be corrected, 
the unfavorable skin, color such as a darkened skin 
color or a greenish skin color is selected by a pointing 
device. The hue of a target picture element correspond- 
ing to the object color and the hue of the neighborhood 
are converted into a desired hue. 
[0005] Japanese Unexamined Patent Publication 
(JP-A) No. H06-1 33329 discloses a technique of cor- 
recting color deviation. Specifically, local color deviation 
is detected for a particular color category, not only the 
skin color, and converted into a representative color of 
the particular color category. 

[0006] Japanese Unexamined Patent Publication 
(JP-A) No. H06-121159 discloses the color correction 
technique taking Into account memory colors memo- 
rized by human beings for color categories such as the 
skin color, the sky, and the green vegetation. 
[0007] On the other hand, Japanese Unexamined 
Patent Publication (JP-A) No. H 10-1 98795 discloses a 
color correction technique of arbitrarily selecting a spe- 
cific hue as an object hue to be corrected and defining a 
characteristic degree representative of the distance 
between the specific hue and the hue of the target pic- 
ture element so that the specific hue alone is subjected 
to color correction. 



[000B] In the conventional color correction tech- 
nique disclosed in JP-A H1 1-1 7969, a dialog with a user 
is executed by the use of a graphical user interface 
(GUI). 

5 [0009] By the use of this technique, it is possible to 
manually select the representative color in an object 
area to be corrected and a color correction parameter. 
This allows flexible and excellent color correction. How- 
ever, in order to process a large amount of image data 

10 in a short time, this technique is inappropriate. This is 
because the processing time and the labor required in 
the processing forms a bottleneck and the result of cor- 
rection would fluctuate In dependence upon the skill of 
an operator. 

is [0010] In order to avoid fluctuation of the result of 
correction due to the difference in skill of the operator 
and the bottleneck by the processing time, color correc- 
tion of a color image must be automatically carried out 
without requiring user's operation. In the above-men- 

20 tioned technique disclosed in JP-A H06-1 33329, a des- 
ignated hue region Is detected and converted into a 
representative constant color. However, the color of the 
natural objects such as skin color, blue sky, or green 
vegetation is not constant Therefore, natural color 

25 reproduction can not be achieved unless a resultant 
color after conversion is given a certain degree of fluctu- 
ation. 

[0011] In the above-mentioned technique disclosed 
in JP-A H06-1 21 159, a single predetermined parameter 

30 is used for each object, e.g., natural matter such as skin 
color, blue sky, and green vegetation, which, however, 
has a wide color distribution under the influence of char- 
acteristics of various input apparatuses and various 
image pickup environments. For color correction of such 

35 object having a wide color distribution, it is necessary to 
more precisely select optimum correction parameters in 
dependence upon the hue, the saturation (shade), and 
the brightness (value) of the color of the object detected 
from an input image. Otherwise, excellent color repro- 

40 duction is impossible. 

[0012] In JP-A H1 0-1 98795, proposal is made of 
the manual color correction by use of the GUI to desig- 
nate the object color and the automatic color correction 
exclusively for the skin color. In the automatic color cor- 

45 recti on, the skin color region is detected by the use of 
RGB threshold values or YCbCr threshold values. How- 
ever, if this technique is applied to images picked up 
under various lighting environments or backgrounds, 
the accuracy in detection of the skin color region is 

so insufficient as far as the RGB threshold values or the 
YCbCr threshold values alone are used. 
[0013] In JP-A H06-121159, the object to be cor- 
rected is detected from the input Image by the use of the 
information of an area of the object region determined 

55 from threshold values for the hue, the saturation, and 
the brightness, like in JP-A H10-19B795. Such informa- 
tion is given as a simple histogram. However, due to the 
similar reason, the accuracy in detection of, for exam- 



2 



3 



EP 1 087 614 A2 



4 



pie, skin color is insufficient 
Summary of the invention; 

[0014] It is therefore an object of this invention to 
provide an automatic color correction apparatus and an 
automatic color correction method capable of realizing 
excellent color correction for natural images picked up 
as input images under various lighting environments, 
even if the information as to the lighting environment of 
the input image, the sensitivity characteristic of an input 
sensor, and the like is unavailable. 
[0015] According to this Invention, there Is provided 
an automatic color correction apparatus which is for car- 
rying out color correction upon a specific object in a 
color image and which comprises an object color infor- 
mation memory for memorizing split hue regions 
obtained by splitting a hue distributable region of the 
specific object as well as a distribution frequency; repre- 
sentative color extracting means for extracting a repre- 
sentative color of the specific object from a given input 
image with reference to the memory content of the 
object color Information memory; a color correction 
parameter memory for memorizing color correction 
parameters assigned to the split hue regions; color cor- 
rection parameter determining means for determining 
from the memory content of the color correction param- 
eter memory an optimum color correction parameter 
adapted to the representative color extracted by the rep- 
resentative color extracting means, and color correction 
processing means for carrying out color correction con- 
version acting only upon a specific hue by the use of the 
optimum color correction parameter determined by the 
color correction parameter determining means. 
[0016] According to this invention, there is also pro- 
vided an automatic color correction method which is for 
carrying out color correction upon a specific object in a 
color image and which comprises the steps of extracting 
a representative color of the specific object from a given 
input image, determining, with reference to the content 
of a color correction parameter memory for memorizing 
color correction parameters assigned to split hue 
regions obtained by splitting a hue distributable region 
of the specific object, an optimum color correction 
parameter adapted to the representative color extracted 
in the preceding step, and carrying out color correction 
conversion acting only upon a specific hue by the use of 
the optimum color correction parameter. 
[0017] According to this invention, there is also pro- 
vided a recording medium storing an automatic color 
correction control program for controlling an automatic 
color correction apparatus to carry out color correction 
upon a specific object in a color image. The automatic 
color correction control program controls the automatic 
color correction apparatus to make the automatic color 
correction apparatus carry out operations of extracting 
from a given input image a representative color of the 
specific object, assigning color correction parameters to 



split hue regions obtained by splitting a hue distributable 
region of the specific object, determining an optimum 
color correction parameter adapted to the representa- 
tive color extracted as mentioned above, and carrying 
5 out color correction conversion acting only upon a spe- 
cific hue by the use of the optimum color correction 
parameter. 

[0018] Specifically, a first automatic color correction 
method according to this invention is for carrying out 

io color correction upon a specific object in a color image 
and comprises the steps of extracting from a given input 
image a representative color of the specific object, 
assigning color correction parameters to split hue 
regions obtained by splitting a hue distributable region 

15 of the specific object, determining an optimum color cor- 
rection parameter adapted to the representative color 
extracted as mentioned above) and carrying out color 
correction conversion acting only upon a specific hue by 
the use of the optimum color correction parameter. 

20 [0019] In a second automatic color correction 
method according to this invention, the step of assign- 
ing the color correction parameters assigns the color 
correction parameters not only to the split hue regions 
of the specific object but also to a plurality of split satu- 

25 ration regions and a plurality of split brightness regions 
obtained by splitting a saturation distributable region 
and a brightness distributable region of the specific 
object, respectively. 

[0020] In a third automatic color correction method 

30 according to this invention, the step of extracting the 
representative color extracts the representative color of 
the specific object by the use of the variance of coordi- 
nate positions in the image in addition to the distribution 
frequency related to a hue, the saturation, or the bright- 

35 ness of the specific object, or a combination thereof. 
[0021] A first automatic color correction apparatus 
is for carrying out color correction of a specific object in 
a color image and comprises an object color information 
memory for memorizing split hue regions obtained by 

40 splitting a hue distributable region of the specific object 
as well as the distribution frequency; representative 
color extracting means for extracting a representative 
color of the specific object from a given input image; a 
color correction parameter memory for memorizing 

45 color correction parameters assigned to the split hue 
regions; color correction parameter determining means 
for determining from the color correction parameter 
memory an optimum color correction parameter 
adapted to the representative color extracted by the rep- 

so resentative color extracting means, end color correction 
processing means for carrying out color correction con- 
version acting only upon a specific hue by the use of the 
optimum color correction parameter determined by the 
color correction parameter determining means. 

55 [0022] A second automatic color correction appara- 
tus according to this invention further comprises an 
extended object color information memory for memoriz- 
ing split regions obtained by splitting a distributable 
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range related to a hue, the saturation, or the brightness 
of the specific object or a combination thereof obtained 
by analyzing a plurality of images preliminarily picked up 
for the specific object under various image pickup envi- 
ronments as well as the distribution frequency. The rep- 5 
resentative color extracting means obtains a histogram 
of the hue, the saturation, or the brightness of the spe- 
cific object in the input image or the combination thereof 
with reference to the split regions related to the hue, the 
saturation, the brightness, or a combination thereof, w 
multiplies the distribution frequency by the histogram, 
and extracts as the representative color of the specific 
object a color present in a region having a maximum 
value as a result of multiplication. 

[0023] In a third automatic color correction appara- is 
tus according to this invention, the representative color 
extracting means extracts the representative color of the 
specific object by the use of the variance of coordinate 
positions in the image in addition to the distribution fre- 
quency related to a hue, the saturation, or the bright- 20 
ness of the specific object or a combination thereof. 
[0024] In a fourth automatic color correction appa- 
ratus according to this invention, the color correction 
parameter memory memorizes the color correction 
parameters assigned not only to the spilt hue regions of 25 
the specific object but also to split saturation regions 
and spilt brightness regions obtained by splitting a satu- 
ration distributable region and a brightness distributable 
region of the specific object, respectively. 
[0025] A fifth automatic color correction apparatus 30 
according to this invention further comprises an input- 
dependent object color information memory for memo- 
rizing with respect to each individual input apparatus 
split regions obtained by splitting a distributable range 
related to a hue, the saturation, or the brightness of the 35 
specific object or a combination thereof as well as the 
distribution frequency, an output-dependent color cor- 
rection parameter memory for holding the color correc- 
tion parameter with respect to each individual output 
apparatus, and input/output apparatus selecting means 40 
for allowing a user to select the type of an input/output 
color image processing apparatus. 
[0026] A sixth automatic color correction apparatus 
according to this invention further comprises a sample 
image memory for memorizing a plurality of kinds of 4s 
sample color images of the specific object, manual color 
correction processing means for allowing a user to cor- 
rect the color of the specific object and color correction 
parameter generating means for generating the color 
correction parameters to be memorized in the color cor- so 
rection parameter memory with reference to the result 
of operation of the manual color correction processing 
means. 

[0027] As described above, the representative color 
of the specific object such as the skin color, the blue sky, ss 
or the green vegetation in an image scene is extracted 
with high accuracy and the optimum color correction 
parameter for the representative color thus extracted Is 



determined. In this manner, it is possible to automati- 
cally carry out excellent color correction upon a signifi- 
cant object (for example, skin color, blue sky, or green 
vegetation) in each of natural images picked up under 
various lighting environments. In other words, It Is pos- 
sible to achieve excellent color correction upon each of 
natural images picked up under various lighting environ- 
ments as an input image, even if the information as to 
the lighting environment of the input image, the sensitiv- 
ity characteristic of an Input sensor, and the like Is una- 
vailable.. 



Prlef pescrjption of the Prawing; 

[0026] 

Fig. 1 is a graph showing a hue distributable region 
and a distribution frequency for an object; 
Fig. 2 is a graph showing the hue distributable 
region in Fig. 1 split into split regions; 
Fig. 3 is a graph showing the change in hue of the 
object before and after color correction; 
Fig. 4 Is a block diagram of a color correction appa- 
ratus according to a first embodiment of this inven- 
tion; 

Rg. 5 shows an example of a GUI used in an object 
selecting unit illustrated in Fig. 4; 
Rg. 6 shows an example of the content of an object 
color information memory illustrated in Fig. 4; 
Rg. 7 shows an example of the content of a color 
correction parameter memory illustrated in Fig. 4; 
Rg. 8 is a flow chart for describing the operation of 
the color correction apparatus illustrated in Fig. 4; 
Rg. 9 is a block diagram of a color correction appa- 
ratus according to a second embodiment of this 
invention; 

Rg. 10 shows an example of the content of an 
extended object color Information memory illus- 
trated in Fig. 9; 

Rg. 1 1 is a flow chart for describing the operation of 
the color correction apparatus illustrated in Fig. 9; 
Rg. 12 is a block diagram of a color correction 
apparatus according to a third embodiment of this 
invention; 

Rg. 13 is a flow chart for describing the operation of 
the color correction apparatus illustrated in Fig. 1 2; 
Rg. 14 is a block diagram of a color correction 
apparatus according to a fourth embodiment of this 
invention; 

Rg. 15 shows an example of a GUI used in an 
input/output apparatus selecting unit illustrated in 
Rg.14; 

Rg. 16 Is a block diagram of a color correction 
apparatus according to a fifth embodiment of this 
invention; and 

Rg. 1 7 shows an example of a GUI used in a man- 
ual color correction processing unit illustrated in 
Rg. 16. 
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Description of the Preferred Embodiments: 

[0029] Now, description will be made of several pre- 
ferred embodiments of this invention with reference to 
the drawings. At first, an automatic color correction 5 
method according to this invention will be described. In 
order to realize automatic color correction upon a spe- 
cific object in an input image, the automatic color cor- 
rection method of this invention requires the steps of (1 ) 
extracting a representative color from a specific object 10 
region in the input image and (2) selecting an optimum 
color correction parameter corresponding to the repre- 
sentative color extracted as mentioned above. 
[0030] In step (1), the representative color of the 
specific object region is automatically extracted from is 
any given input image. In this event, the input image 
contains an image which cannot be identified for its 
source, I.e., an Input apparatus such as a digital cam- 
era, a scanner, an image database on an internetwork, 
and so on. This means that the sensor characteristic of 20 
the Input apparatus cannot be identified. Therefore, it is 
difficult to estimate a lighting color and a substance 
color (or a body color) in an image scene as key factors 
of color correction. 

[0031] In order to extract the representative color 25 
from the specific object region in each of various input 
images, preparation is preliminarily made of a distribut- 
able range and a distribution frequency for each of a 
hue, a saturation, and a brightness of a color obtained 
from the specific object region in each of image data of 30 
the specific object picked up in various lighting environ- 
ments, on various backgrounds, and by various input 
apparatuses. 

[0032] The distributable range and the distribution 
frequency related to the hue, the saturation, and the 35 
brightness of the color of the specific object region are 
similar to a recognition dictionary in pattern recognition. 
Preparation of the distributable range and the distribu- 
tion frequency requires manual operation. However, if 
once prepared, it is enough. Specifically, upon actually 40 
extracting the representative color of the specific object 
from the input image, use is made of the distributable 
range and the distribution frequency related to the hue, 
the saturation, the brightness of the specific object 
region. In this manner, the representative color of the 45 
specific object can be automatically extracted. 
[0033] Referring to Fig. 1, a given specific object 
has a distributable range and the distribution frequency 
related to the hue. In the figure, a region between a 
lower limit hue A and an upper limit hue B is a hue range so 
over which the specific object can be distributed, i.e., a 
hue distributable range. An ordinate represents the dis- 
tribution frequency of the specific object in the distribut- 
able range. In Fig. 1 , the distribution frequency and the 
distributable range are directed to the hue alone. How- 55 
ever, the distribution frequency and the distributable 
range can also be prepared for a saturation component 
and a brightness component Furthermor , it Is possible 



to prepare the distributable range and the distribution 
frequency in a three-dimensional space of integration of 
three components, i.e., the hue, the saturation, and the 
brightness. 

[0034] Referring to Fig. 2, the hue distributable 
range in Fig. 1 is split into a plurality of regions, n in 
number, at a particular split interval. The spilt interval 
may be constant or variable. In the example illustrated in 
Fig. 2, the hue distributable range of the specific object 
is divided into eleven regions. In the following, those 
regions obtained by dividing the hue distributable range 
will be referred to as spilt hue regions. 
[0035] Calculation is made of the distribution fre- 
quency R H (i) (i being an integer between 1 and n, both 
inclusive). The suffix H represents the hue. The distribu- 
tion frequency Rn(a) for a particular split hue region a is 
calculated by: 

R H (a) = b/a11, (1) 

where b represents the number of samples present In 
the particular split hue region a and a1 1 represents the 
total number of samples. Similarly, calculation can also 
be made of the distribution frequency R s (j) for the satu- 
ration and the distribution frequency R v (k) for the bright- 
ness. 

[0036] Next, description will be made of a technique 
of extracting a region containing the representative 
color of the specific object from the input image picked 
up under a given lighting environment. In this technique, 
use is made of the distributable range of the specific 
object and the distribution frequency in each of the spilt 
regions mentioned above. For simplicity, description will 
be directed to the hue. 

[0037] Calculation is made of the hue, the satura- 
tion, and the brightness of each picture element of the 
input image. Herein, the hue, the saturation, and the 
brightness may be calculated as H (Hue), S (Shade), 
and V (Value) in the following equations widely used in 
color image processing, respectively, but are not 
restricted thereto. The shade and the value correspond 
to the saturation and the value, respectively. 

V = Max (R, G, B) (2) 

when V = 0 S = 0 

when V > 0 S = (V - Mln(R, G, B))/V 

when S = 0 H = 0 

when V = R H = 60(G - B)/SV 

when V = G H = 60(2 + (B - R)/SV) 

when V = B H = 60(4 + (R - G)/SV) 

when H < 0 H = H + 360 
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[0038] For each picture element, judgment is made 
about whether or not its hue fails within the hue distrib- 
utable range of the specific object If it is judged to fall 
within the range, detection is then made of one of the 
split hue regions, n in number to which the hue belongs. 
[0039] The above-mentioned operation is carried 
out far all of the picture elements contained in the input 
image to prepare a histogram HIT H (l) in each of all split 
hue regions. Herein, I is an integer between 1 and n, 
both inclusive, and represents an l-th region among the 
the split hue regions, n in number. Then, calculation is 
made of a representative color candidate index RP(i) in 
each split hue region as: 

RP(i) = HIT H (i)xR H (i), (3) 

where i is an integer between 1 and n, both inclusive. 
[0040] In this case, one of the split hue regions 
which has a greatest value of the representative color 
candidate index RP is determined as a selected split 
hue region of the specific object in the input image. As 
the representative color of the specific object in the 
input image, use may be made of average RGB values 
of those picture elements present within the selected 
split hue region. Alternatively, each of the split hue 
regions may be preliminarily assigned with predeter- 
mined RGB values which would be used as the repre- 
sentative color. The predetermined RGB values may be 
obtained as average RGB values of those colors 
present in each split region by the use of the image data 
set used in obtaining the distributable range and the dis- 
tribution frequency related to each of the hue, the satu- 
ration, and the brightness of the specific object. 
[0041 ] The above-mentioned technique is a method 
of extracting the representative color of the specific 
object with reference to the color information atone. In 
order to further improve the accuracy in extracting the 
representative color, introduction is made of the vari- 
ance of coordinate positions in the image with respect to 
the picture elements present in each split hue region. 
For example, it is assumed that the picture elements, K 
in number, are present within a split hue region a. Each 
of the picture elements has an XY coordinate (XI, Yi) (I 
being an integer between 1 and K, both inclusive) in the 
image. For those picture elements, an average coordi- 
nate (Xc, Yc) is calculated by: 

Xc = ZXi/K and (4) 

Yc = ZYi/K, 

where £ represents the total sum for i = 1 to K. 
[0042] The variance Va of the coordinate positions 
of the picture elements present within the spilt hue 
region a is given by: 

Va = X((Xc - Xi) 2 + ( Yc - Yi) 2 /K, (5) 



where r represents the total sum for i = 1 to K. 
[0043] The difference between the variance Va and 
the standard variance SV of the specific object is intro- 
duced into the evaluation of the split hue region. The 

5 standard variance of the specific object can be calcu- 
lated by the use of the image data set used in obtaining 
the distributable range and the distribution frequency 
related to each of the hue, the saturation, and the bright- 
ness of the specific object 

w [0044] The evaluation of the split hue region may be 
carried out in various manners. For example, a thresh- 
old value is determined for the difference between the 
variance Va and the standard variance SV. If the differ- 
ence in variance exceeds the threshold value, the 

is region in question is discarded. Then, evaluation is car- 
ried out for the difference between the standard vari- 
ance and the variance of another split hue region having 
a second greatest value of the representative color can- 
didate index RR Alternatively, definition is made of a 

20 modified representative color candidate index RP 1 given 
by: 

RP'(o) = HIT H (o) x R H (o) x C D(o) , (6) 

25 where C is a real number constant in the range of 0 < C 
< 1 and D(o) is a real number in the range of D(o) s 0.0. 
In the modified representative color candidate index 
RP', an absolute value D(o) of the difference between 
the standard variance SV and the variance Vo of the 

30 specific object in a split hue region o is introduced. A 
particular spilt hue region having the maximum value of 
the modified representative color candidate index RP* is 
determined as a hue region which the specific object in 
the input image has. 

35 [0045] The representative color is determined as an 
average of the colors present in the split hue region 
which is selected taking into account the variance of the 
coordinate positions of the specific object also. 
[0046] Next, description will be made of the tech- 

40 nique of selecting an optimum color correction parame- 
ter corresponding to the representative color extracted 
In the step (2). For a plurality of image data sets includ- 
ing the specific object, color correction Is preliminarily 
carried out by manual operation so that the specific 

45 object has an excellent color. The color correction by 
the manual operation may be carried out by the color 
correction technique acting only on a specific hue as 
disclosed in JP-A H1 0-1 98795 and the technique utiliz- 
ing the GUI. 

so [0047] Referring to Fig. 3, the variation in hue of the 
specific object before and after the color correction is 
averaged for each of the split hue regions. The start 
point and the end point of each arrow represent a hue 
before correction and a hue after correction, respec- 

55 tively. 

[0048] The color correction parameter for generat- 
ing the variation is described in a color correction 
parameter memory for each of the split hue regions. 
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Thereafter, calculation is made of the hue of the repre- 
sentative color of the specific object extracted from the 
input image. The color correction parameter assigned to 
the split hue region where the hue is present is read 
from the color correction parameter memory to be 5 
determined. 

[0049] The foregoing description is directed to the 
color correction technique with reference to the hue 
alone. It will readily be understood that the color correc- 
tion with high accuracy can be achieved by incorporat- io 
ing the information of the saturation and the brightness 
by the use of the similar technique. 
[0050] Referring to Fig. 4, a color correction appa- 
ratus according to a first embodiment of this invention 
comprises an object selecting unit 1 1 for selecting an is 
object to be subjected to color correction, an object 
color information memory 12 for memorizing the distrib- 
utable region and the distribution frequency with respect 
to the hue for each specific object preliminarily 
assumed, a representative color extracting unit 1 3 for 20 
extracting the representative color of the specific object 
from the input image, a color correction parameter 
memory 1 4 for memorizing color correction parameters, 
a color correction parameter determining unit 15 for 
determining from the content of the color correction 25 
parameter memory 14 an optimum color correction 
parameter corresponding to the representative color 
extracted from the input image by the representative 
color extracting unit 13, and a color correction process- 
ing unit 1 6 for carrying out color correction exclusively 30 
upon the representative color extracted as mentioned 
above and neighborhood colors therearound. 
[0051] Referring to Figs. 4 through 8, description 
will be made of the structure and the operation of the 
color correction apparatus 1 . The operation illustrated in 35 
Fig. 8 is achieved when each component of the color 
correction apparatus 1 executes a program stored in a 
control memory (not shown). The control memory may 
comprise a ROM (Read Only Memory), an IC (Inte- 
grated Circuit) memory, or the like. 40 
[0052] The object selecting unit 11 has a function of 
allowing a user to easily select the object to be sub- 
jected to color correction. Referring to Fig. 5, the object 
selecting unit 11 uses a GUI inviting the selection 
among three options, i.e., skin color, green vegetation, 45 
and blue sky, as the object of color correction, 
[0053] The object selecting unit 11 may be 
arranged so that a plurality of objects can be selected, 
provided that the distributable regions of the respective 
objects in a color space are not overlapped. In Fig. 5, so 
the skin color is selected. 

[0054] The representative color extracting unit 1 3 at 
first reads from the object color information memory 12 
the distributable region of the hue of the object selected 
by the object selecting unit 1 1 and the distribution fre- ss 
quency in the spilt hue region. 

[0055] In the object color information memory 12, 
the Information about a plurality of objects A, B, and C is 



described. Referring to Fig. 5, the information about the 
object A includes a split hue region number, upper and 
lower limit values of the hue for designating each split 
hue region, and th frequency with respect to the object 
A. Although not illustrated In the figure, the object color 
information memory 12 includes the description of the 
information of the objects B and C, similar to that of the 
object A. 

[0056] The representative color extracting unit 13 
calculates the hue for each of all picture elements of the 
input image, prepares the histogram for each split hue 
region, calculates the representative color candidate 
index RP In each split hue region by the use of Equation 
(3), and selects a particular spirt hue region having the 
maximum value as a region occupied by the object A in 
the input image. Furthermore, the representative color 
extracting unit 13 determines, as the representative 
color of the object A in the input Image, the average 
RGB value of the colors present in the split hue region 
in question (step S1 in Fig. 8). 

[0057] The color correction parameter determining 
unit 15 determines from the memory content of the color 
correction parameter memory 1 4 the optimum color cor- 
rection parameter corresponding to the representative 
color extracted by the representative color extracting 
unit 13. Specifically, with reference to the memory con- 
tent of the color correction parameter memory 14 mem- 
orizing the color correction parameters assigned to the 
split hue regions obtained by splitting the hue distributa- 
ble region of the specific object, the color correction 
parameter determining unit 15 determines the optimum 
color correction parameter corresponding to the repre- 
sentative color extracted by the representative color 
extracting unit 13 (step S2 in Fig. 8). 
[0058] The color correction parameter memory 14 
memorizes the color correction parameters for the 
objects A, B, and C. Referring to Fig. 7, the description 
of the color correction parameters about the object A 
alone is illustrated. Although not shown in the figure, the 
content similar to the object A is also described in the 
color correction parameter memory 14 for each of the 
objects B and C. For example, it is assumed that the 
representative color extracting unit 13 determines that 
the hue of the object A is a split hue region #2. Then, the 
color correction parameters (P2, Q2, R2, S2) are deter- 
mined. 

[0059] The color correction processing unit 1 6 car- 
ries out, upon the input Image, the color correction only 
for the representative color of the object and the neigh- 
borhood colors therearound. Specifically, the color cor- 
rection processing unit 16 carries out color correction 
conversion acting only on the specific hue by the use of 
the optimum color correction parameter (step S3 in Fig. 
B). 

[0060] The above-mentioned color correction may 
be carried out by the color correction technique dis- 
closed, for example, in JP-A H10-198795. If the color 
correction technique disclosed In JP-A H10-198795 is 
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used, the color correction for the input RGB is repre- 
sented by: 

(R\ G\ B*) = (R, G, B) + hx x (a1 , a2, a3), (7) 

where (R, G, B) is a set of given RGB values in the input 
image, (R\ G\ B') is a set of RGB values after correc- 
tion, (a1 , a2, a3) is a set of an R correction amount, a G 
correction amount, and a B correction amount, and hx 
represents as a characteristic degree the distance 
between a center color (Rc, Gc, Be) of correction and 
the given RGB values (R, G t B). 
[0061] The distance hx Is given by: 

hx = [pos( m - 1 Hue - hi I )/m] x si x vl, (8) 

where pos(x) = 0 when x < 0 and pos(x) = x when xaO, 
m represents an allowable hue angle, Hue is a Hue 
value of HSV values (Hue, Sat, Val) calculated from the 
RGB values of the object to be corrected, and hi, si, vl 
are HSV values of the given RGB values. 
[0062] The representative color extracted by the 
representative color extracting unit 13 Is used as a 
center color of color correction. The color correction 
parameters (P2, Q2, R2, S2) determined by the color 
correction parameter determining unit 1 5 represent the 
R correction amount, the G correction amount, the B 
correction amount, and the allowable hue angle, 
respectively. 

[0063] The foregoing is directed to the color correc- 
tion apparatus 1 addressed to the hue alone. It will be 
understood that an improved color correction apparatus 
of high accuracy can be achieved by extending the 
object color information memory 12 of the color correc- 
tion apparatus 1 to incorporate saturation and bright- 
ness components in addition to the hue. 
[0064] Referring to Fig. 9, a color correction appa- 
ratus 2 according to a second embodiment of this inven- 
tion is similar in structure to the color correction 
apparatus 1 in Fig. 4 according to the first embodiment 
except that the object color information memory 12 is 
replaced by an extended object color information mem- 
ory 17. Similar parts are designated by like reference 
numerals and perform similar operations. Specifically, 
the color correction apparatus 2 takes into account not 
only the hue but also the saturation and the brightness 
components. 

[0065] Referring to Fig. 10, the extended object 
color information memory 1 7 comprises a split informa- 
tion section and a frequency information section. The 
split information section includes the description of 
upper and lower limit values for each of the hue, the sat- 
uration, and the brightness components In order to split 
the distributable region of the object. The frequency 
information section includes the description of the fre- 
quency in each split region. 

[0066] In Fig. 10, the description about the object A 
alone in the extended object color information memory 



17 is illustrated. However, the extended object color 
information memory 1 7 includes similar description for 
each of the objects B and C. In Fig. 10, the distributable 
region of the object A is split into N for the hue, M for the 

5 saturation, and L for the brightness. Thus, the total 
number of the split regions is equal to N x M x L. 
[0067] Referring to Fig. 1 1 in addition to Figs. 9 and 
10, the operation of the color correction apparatus 2 
according to the second embodiment will be described. 

w The operation illustrated in Fig. 11 is achieved when 
each component of the color correction apparatus 2 
executes a program stored in a control memory (not 
shown). The control memory may comprise a ROM, an 
iC memory, or the like. 

is [0068] In a manner similar to that described in con- 
junction with the color correction apparatus 1 , the repre- 
sentative color extracting unit 13 prepares the 
histogram for each split region split by the hue, the sat- 
uration, and the brightness, calculates the representa- 

20 tive color candidate index RP in each split region, and 
selects a particular split region having the maximum 
value as a split region which the object has. Further- 
more, the representative color extracting unit 13 deter- 
mines, as the representative color of the object in the 

25 input image, the average RGB values of the colors 
present in the split hue region (step S1 1 in Fig. 1 1 ). 
[0069] The color correction parameter determining 
unit 1 5 determines from the memory content of the color 
correction parameter memory 14 the optimum color cor- 

30 recti on parameters corresponding to the representative 
color extracted by the representative color extracting 
unit 13. Specifically, with reference to the memory con- 
tent of the color correction parameter memory 1 4 mem- 
orizing the color correction parameters assigned to the 

35 split regions, N x M x L in total, obtained by splitting the 
distributable region of the specific object related to the 
hue, the saturation, and the brightness into N for the 
hue, M for the saturation, and L for the brightness, the 
color correction parameter determining unit 15 deter- 

40 mines the optimum color correction parameters corre- 
sponding to the representative color extracted by the 
representative color extracting unit 13 (step S12 in Fig. 
11). 

[0070] The color correction processing unit 16 car- 
45 ries out, upon the input image, the color correction only 
for the representative color of the object and the neigh- 
borhood colors therearound. Specifically, the color cor- 
rection processing unit 1 6 carries out color correction 
conversion acting only upon the specific hue by the use 
so of the optimum color correction parameters (step S1 3 in 
Fig. 11). The color correction parameter memory 14 in 
the color correction apparatus 2 holds the color correc- 
tion parameters for the split regions equal In number to 
N x M x L. 

55 [0071 ] Referring to Fig. 1 2, a color correction appa- 
ratus 3 according to a third embodiment of this invention 
is similar in structure to the color correction apparatus 2 
in Fig. 9 according to the second embodiment except 
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that the representative color extracting unit 13 is 
replaced by a histogram preparing unit 18, a variance 
calculating unit 19, an object region variance memory 
20, and a representative color determining unit 21 . Sim- 
ilar parts are designated by like reference numerals and 
perform similar operations. Thus, upon extracting the 
representative color of the object from the input image, 
the color correction apparatus 3 takes into account not 
only the color information of the object but also the var- 
iance of coordinate positions on the image. 
[0072] Referring to Fig. 1 3 in addition to Fig. 12, the 
operation of the color correction apparatus 3 according 
to the third embodiment will be described. The opera- 
tion illustrated in Fig. 13 is realized when each compo- 
nent of the color correction apparatus 3 executes a 
program stored in a control memory (not shown). The 
control memory may comprise a ROM, an IC memory, 
or the like. 

[0073] At first, the histogram preparing unit 18 pre- 
pares a histogram in each split region of the object des- 
ignated by the extended object color information 
memory 1 7 and memorizes image coordinates of those 
colors hitting the split region. The variance calculating 
unit 19 calculates an average coordinate position in 
each split region by the use of Equation (4) and calcu- 
lates the variance of coordinate positions in each split 
region by the use of Equation (5). 
[0074] Next, in order to calculate the modified rep- 
resentative color candidate index RP', the representa- 
tive color determining unit 21 reads the standard 
variance of the object from the object region variance 
memory 20, calculates the difference between the 
standard variance and the variance of the coordinate 
positions in each split region, and calculates the modi- 
fied representative color candidate index RP' in each 
split region by the use of Equation (6). A particular split 
region having the maximum value is selected as a hue 
region occupied by the specific object in the input 
image. In the manner similar to that carried out by the 
representative color extracting unit 13, the representa- 
tive color determining unit 21 determines as the repre- 
sentative color the average RGB values of the colors 
present in the particular split region selected as men- 
tioned above (step S21 in Fig. 13). 
[0075] The color correction parameter determining 
unit 1 5 determines from the memory content of the color 
correction parameter memory 1 4 the optimum color cor- 
rection parameters corresponding to the representative 
color extracted by the representative color extracting 
unit 13. Specifically, with reference to the memory con- 
tent of the color correction parameter memory 14 mem- 
orizing the color correction parameters assigned to the 
split regions, N x M x L in total, obtained by splitting the 
distributable region of the specific object related to the 
hue, the saturation, and the brightness into N for the 
hue, M for the saturation, and L for the brightness, the 
color correction parameter determining unit 15 deter- 
mines the optimum color correction parameters corre- 



sponding to the representative color extracted by the 
representative color extracting unit 13 (step S22 in Fig. 
13). 

[0076] The color correction processing unit 1 6 car- 
5 ries out, upon the input image, the color correction only 
for the representative color of the object and the neigh- 
borhood colors therearound. Specifically, the color cor- 
rection processing unit 16 carries out color correction 
conversion acting only upon the specific hue by the use 
10 of the optimum color correction parameters (step S23 in 
Fig. 13). 

[0077] Referring to Fig. 14, a color correction appa- 
ratus 4 according to a fourth embodiment of this inven- 
tion comprises an object selecting unit 11, a 

15 representative color extracting unit 1 3, a color correc- 
tion parameter determining unit 15, a color correction 
processing unit 16, an input-dependent object color 
Information memory 22, an output-dependent correc- 
tion parameter memory 23, and an input/output appara- 

20 tus designating unit 24. 

[0078] Generally, color image input/output appara- 
tuses are different in color characteristic from one 
another. If it is known that the Input image Is acquired by 
an input apparatus preliminarily assumed, the distribu- 

25 tion region of the object can be restricted to some 
extent. In case an output apparatus for an output image 
after color correction can be identified, color correction 
can further be improved by preparing the color correc- 
tion parameters such that the color correction into the 

30 optimum color is achieved for the output image pro- 
duced by the output apparatus as identified. 
[0079] Referring to Fig. 1 5, the input/output appara- 
tus designating unit 24 uses a GUI illustrated in the fig- 
ure to provide a function of allowing a user to select an 

35 input apparatus or an output apparatus. The GUI illus- 
trated in Fig. 15 allows the selection of three kinds of 
input apparatuses A, B, and C and three kinds of output 
apparatuses D, E, and F. As far as the Input and the out- 
put apparatuses can be selected, any other appropriate 

40 arrangement is bapplicable. 

[0080] The input-dependent object color informa- 
tion memory 22 memorizes the distributable region of 
the object and the distribution frequency as illustrated in 
Fig. 6 or 1 0 for each kind of the models A, B, and C as 

45 the input apparatuses assumed by the coior correction 
apparatus. 

[0081] The input/output apparatus designating unit 
24 reads from the Input-dependent object color informa- 
tion memory 22 the distributable region of the object 

so and the distribution frequency corresponding to the 
selected input apparatus and delivers the distributable 
region of the object and the distribution frequency to the 
representative color extracting unit 13. The representa- 
tive color extracting unit 13 extracts the representative 

55 color with reference thereto. 

[0082] The output-dependent color correction 
parameter memory 23 memorizes the color correction 
parameters illustrated In Fig. 7 for each split region of 
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the object with respect to each of the output appara- 
tuses D, E, and F assumed by the color correction appa- 
ratus. 

[0083] The input/output apparatus designating unit 

24 reads from the output-dependent color correction s 
parameter memory 23 the color correction parameters 

of the object corresponding to the output apparatus des- 
ignated by the user and sends the color correction 
parameters of the object to the color correction parame- 
ter determining unit 15. From the color correction 10 
parameters delivered from the input/output apparatus 
designating unit 24, the color correction parameter 
determining unit 1 5 determines the optimum color cor- 
rection parameters adapted to the representative color 
extracted by the representative color extracting unit 13. is 
In the above-mentioned manner, the color correction 
apparatus 4 realizes color correction of high accuracy in 
case the input/output apparatus can be identified. 
[0084] Referring to Fig. 1 6, a color correction appa- 
ratus 5 according to a fifth embodiment of this invention 20 
is similar in structure to the color correction apparatus 1 
in Fig. 4 according to the first embodiment except that a 
manual color correction processing unit 25, a sample 
image memory 26, and a color correction parameter 
generating unit 27. Similar parts are designated by like 25 
reference numerals and perform similar operations. 
[0085] Generally, for the specific object such as skin 
color, green vegetation, or the blue sky in a color Image, 
a favorable color or a desired color is present However, 
the favorable color for each object is more or less differ- 30 
ent depending on the taste of the user. Therefore, the 
color correction apparatus 5 according to the fifth 
embodiment further comprises the manual color correc- 
tion processing unit 25, the sample image memory 26, 
and the color correction parameter generating unit 27 in 35 
order to meet the demand of each user in addition to the 
structure of the color correction apparatus 1 . 
[0086] The manual color correction processing unit 

25 has a color correction algorithm similar to that of the 
color correction processing unit 1 6 but the color correc- 40 
tion parameters are given by the user. Herein, the sam- 
ple image memory 26 memorizes a plurality of sample 
Images of the assumed object so that the color of the 
object is widely distributed. 

[0087] The manual color correction processing unit 45 
25 displays the sample images of the object supplied 
from the sample image memory 26 and provides the 
GUI for the user to execute color correction of the object 
[0088] Referring to Fig. 1 7, the manual color correc- 
tion processing unit 25 in Fig. 16 provides the GUI in so 
which the object A is selected as the object to be cor- 
rected. The counter shows that a first image of the sam- 
ple images, N In number, is displayed. 
[0089] At the center of the GUI, the images before 
and after correction are displayed. In the image before ss 
color correction, it is shown that only the object A 
depicted by an arrow is subjected to color correction. 
The representative color of the object A in the sample 



image is designated by the user with a pointing device 
such as a mouse. The user adjusts th color correction 
parameters 1 through 4 until he is satisfied with the 
image after correction. 

[0090] After completion of the first image, the user 
pushes a "next image display" button at a right upper 
portion of the GUI. The information of the representative 
color and the color correction parameters selected for 
each sample image is held. After completion of manual 
color correction for all of the sample images, all sets of 
the representative color and the color correction param- 
eters designated by the user are delivered to the color 
correction parameter generating unit 27. 
[0091] Supplied from the object color information 
memory 12 with the information of the split regions of 
the object, the color correction parameter generating 
unit 27 selects, for each split region, the representative 
color present In the region and the color correction 
parameters thereat among all sets of the representative 
color and the color correction parameters supplied from 
the manual color correction processing unit 25, calcu- 
lates the average value of the color correction parame- 
ters, and describes the average value In the 
corresponding split region of the color correction param- 
eter memory 14. 

[0092] The representative color extracting unit 13 
reads from the object color information memory 12 the 
distributable region of the hue of the selected object 
selected by the object selecting unit 1 1 and the distribu- 
tion frequency in the split hue region to extract the rep- 
resentative color. 

[0093] From the memory content of the color cor- 
rection parameter memory 14, the color correction 
parameter determining unit 15 determines the optimum 
color correction parameters adapted to the representa- 
tive color extracted by the representative color extract- 
ing unit 13. The color correction processing unit 16 
carries out, upon the input image, color correction act- 
ing only upon the representative color of the object and 
the neighborhood colors therearound. As described 
above, the color correction apparatus 5 can carry out 
color correction upon the object in the input image in 
response to the taste of the user. 
[0094] As described above, the color correction is 
automatically carried out for the specific object in the 
color image so that the color of the significant object, 
such as skin color, blue sky, and green vegetation con- 
tained in the natural image can be automatically cor- 
rected into a favorable or the desired color without 
requiring masking operation or the dialog with the user. 
Furthermore, it is possible to provide the apparatus with 
the characteristic color reproducibility by incorporating 
the operation of this Invention Into the main body of the 
color image processing apparatus or the driver soft- 
ware. 

[0095] As described above, according to this inven- 
tion, the automatic color correction apparatus for carry- 
ing out color correction upon the specific object in the 
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color image is controlled to perform the steps of extract- 
ing the representative color of the specific object from 
the given input image, assigning the color correction 
parameters to the split hue regions obtained by dividing 
the hue distributable region of the specific object, deter- 5 
mining the optimum correction parameters adapted to 
the extracted representative color, and carrying out 
color correction conversion acting only upon the specific 
hue by the use of the optimum color correction parame- 
ters. In this manner, for each of the natural Images 10 
picked up under the various lighting environments as 
the input image, excellent color correction can be real- 
ized without having the information as to the lighting 
environment of the input image and the sensor charac- 
teristics of the Input sensor. J5 

Claims 

1. An automatic color correction apparatus for carry- 
ing out color correction upon a specific object in a 20 
color image, said apparatus comprising an object 
color information memory for memorizing split hue 
regions obtained by splitting a hue distributable 
region of the specific object as well as a distribution 
frequency; representative color extracting means 2s 
for extracting a representative color of the specific 
object from a given input image with reference to 

the memory content of the object color Information 
memory; a color correction parameter memory for 
memorizing color correction parameters assigned 30 
to the split hue regions; color correction parameter 
determining means for determining from the mem- 
ory content of the color correction parameter mem- 
ory an optimum color correction parameter adapted 
to the representative color extracted by the rep re- 35 
sentative color extracting means, and color correc- 
tion processing means for carrying out color 
correction conversion acting only upon a specific 
hue by the use of the optimum color correction 
parameter determined by the color correction 40 
parameter determining means. 

2. An automatic color correction apparatus as claimed 
in claim 1, further comprising: 

45 

an extended object color information memory 
for memorizing split regions obtained by split- 
ting a distributable range related to a hue, the 
saturation, or the brightness of the specific 
object or a combination thereof obtained by so 
analyzing a plurality of images preliminarily 
picked up for the specific object under various 
image pickup environments as well as the dis- 
tribution frequency; 

said representative color extracting means 55 
obtaining a histogram of the hue, the satura- 
tion, or the brightness of the specific object in 
the Input image or the combination thereof with 



reference to the split regions related to the hue, 
the saturation, the brightness, or a combination 
thereof, multiplying the distribution frequency 
memorized in said extended object color infor- 
mation memory by the histogram, and extract- 
ing as the representative color of the specific 
object a color present in a region having a max- 
imum value as a result of multiplication. 

3. An automatic color correction apparatus as claimed 
in claim 1 , wherein the representative color extract- 
ing means extracts the representative color of the 
specific object by the use of the variance of coordi- 
nate positions in the image in addition to the distri- 
bution frequency related to a hue, the saturation, or 
the brightness of the specific object or a combina- 
tion thereof. 

4. An automatic color correction apparatus as claimed 
in claim 1 wherein the color correction parameter 
memory memorizes the color correction parame- 
ters assigned not only to the split hue regions of the 
specific object but also to spilt saturation regions 
and split brightness regions obtained by splitting a 
saturation distributable region and a brightness dis- 
tributable region of the specific object, respectively. 

5. An automatic color correction apparatus as claimed 
in claim 1 , further comprising an input-dependent 
object color information memory for memorizing 
with respect to each individual input apparatus split 
regions obtained by splitting a distributable range 
related to a hue, the saturation, or the brightness of 
the specific object or a combination thereof as well 
as the distribution frequency, an output-dependent 
color correction parameter memory for holding the 
color correction parameter with respect to each 
individual output apparatus, and input/output appa- 
ratus selecting means supplied from the outside 
with the type of an input/output color image 
processing apparatus for reading corresponding 
information from said input-dependent object color 
Information memory and said output-dependent 
color correction parameter memory. 

6. An automatic color correction apparatus as claimed 
in claim 1, further comprising a sample image 
memory for memorizing a plurality of kinds of sam- 
ple color images of the specific object, manual color 
correction processing means responsive to exter- 
nal instruction for entering color information of said 
color images memorized in said sample image 
memory, and color correction parameter generating 
means for generating the color correction parame- 
ters to be memorized in the color correction param- 
eter memory with reference to the information 
supplied from the manual color correction process- 
ing means. 



11 



21 



EP1 087 614 A2 



7. An automatic color correction method for carrying 
out color correction upon a specific object in a color 
image, said method comprising the steps of extract- 
ing a representative color of the specific object from 

a given input image, determining, with reference to 5 
the content of a color correction parameter memory 
for memorizing color correction parameters 
assigned to split hue regions obtained by splitting a 
hue distributable region of the specific object, an 
optimum color correction parameter adapted to the w 
representative color extracted in the preceding 
step, and carrying out color correction conversion 
acting only upon a specific hue by the use of the 
optimum color correction parameter. 

15 

8. An automatic color correction method as claimed in 
claim 7, wherein the color correction parameter 
memory memorizes the color correction parame- 
ters assigned not only to the split hue regions of the 
specific object but also to split saturation regions 20 
and split brightness regions obtained by splitting a 
saturation distributable region and a brightness dis- 
tributable region of the specific object, respectively. 

9. An automatic color correction method as claimed in 25 
claim 7, wherein the step of extracting the repre- 
sentative color extracts the representative color of 

the specific object by the use of the variance of 
coordinate positions in the image in addition to the 
distribution frequency related to a hue, the satura- 30 
tion, or the brightness of the specific object, or a 
combination thereof. 

10. A recording medium storing an automatic color cor- 
rection control program for controlling an automatic 35 
color correction apparatus to carry out color correc- 
tion upon a specific object in a color image, said 
automatic color correction control program control- 
ling the automatic color correction apparatus to 
make the automatic color correction apparatus 40 
carry out operations of extracting from a given input 
image a representative color of the specific object, 
assigning color correction parameters to split hue 
regions obtained by splitting a hue distributable 
region of the specific object, determining an opti- 45 
mum color correction parameter adapted to the rep- 
resentative color extracted as mentioned above, 
and carrying out color correction conversion acting 
only upon a specific hue by the use of the optimum 
color correction parameter. so 
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